1. Field of the Invention
The present invention relates to an image processing technique for performing an image restoration process to correct (reduce) a degraded component of an image produced by image capturing through an image capturing optical system.
2. Description of the Related Art
Captured images produced by image pickup apparatuses such as digital cameras include a blur component that is an image degradation component caused by various aberrations of an image capturing optical system (hereinafter simply referred to as “an optical system”) such as spherical aberration, comatic aberration, field curvature and astigmatism. Such a blur component is generated because a light flux emitted from one point of an object forms an image with some divergence on an image pickup surface of an image sensor, the light flux being normally converged at one point if there is no aberration or diffraction.
The blur component herein is optically expressed as a point spread function (PSF), and is different from blur caused by defocusing. Moreover, color blur in a color image generated due to longitudinal chromatic aberration, chromatic spherical aberration or chromatic comatic aberration of the optical system can be said to be a difference between blurring degrees of respective light wavelengths. Moreover, lateral color shift caused by chromatic aberration of magnification of the optical system can be said to position shift or phase shift due to difference of image capturing magnifications for respective light wavelengths.
Fourier transform of the point spread function (PSF) provides an optical transfer function (OTF) showing frequency component information on aberration and being expressed by a complex number. An absolute value of the optical transfer function (OTF), that is, an amplitude component is called a modulation transfer function (MTF), and a phase component is called a phase transfer function (PTF). The MTF and PTF respectively show frequency characteristics of amplitude and phase components of image degradation due to aberration. In the following description, the phase component is expressed as a phase angle by the following expression where Re(OTF) and Im(OTF) respectively represent a real part and an imaginary part of the OTF.PTF=tan−1[Im(OTF)/Re(OTF)]
As described above, the optical transfer function (OTF) of the optical system provides degradation to the amplitude and phase components of the image, so that the degraded image includes, at each point of the object, asymmetric blur like comatic aberration.
Moreover, the chromatic aberration of magnification is generated by separately capturing images of color components such as R, G and B, whose imaging positions are mutually shifted due to differences of imaging magnifications for the respective light wavelengths, depending on spectral characteristics of the image pickup apparatus. The imaging magnification differences for the respective light wavelengths cause not only the shift of the imaging positions among the color components, but also shift of imaging positions of respective wavelengths in each of the color components, that is, image spread due to the phase shift. Thus, although, exactly, the chromatic aberration of magnification is not mere a parallel color shift, the following description treats the color shift as being equivalent to the chromatic aberration of magnification.
As a method for correcting such degradation of the amplitude component (MTF) and degradation of the phase component (PTF) in the degraded image (input image), there is known a method using information on the optical transfer function (OTF) of the optical system. This method is referred to as “image restoration” or “image recovery”, and a process for correcting the degraded image by using the information on the optical transfer function (OTF) of the optical system is hereinafter referred to as “an image restoration process (or simply, a restoration process)”. As one of the image restoration processes, a convolution method that performs convolution of an image restoration filter having an inverse characteristic to the optical transfer function on the input image in a real space. Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2005-509333 discloses a method that performs a restoration process by storing and using filter coefficients for correcting degradation of an image.
When performing image capturing with focusing on a main object, images of other objects located at different distances become defocused images. Therefore, when performing the restoration process on such defocused images, it is necessary to use image restoration filters corresponding to positions of the respective objects (that is, defocus amounts). However, using such image restoration filters corresponding to the defocus amounts increases amount of data of the image restoration filter to be provided (that is, to be stored). Moreover, performing the image restoration process also on the defocused image increases processing time. Thus, Japanese Patent Laid-Open No. 2011-044825 discloses a method that performs an image restoration process only on an in-focus area of a captured image to increase a processing speed.
Since the image restoration filter to be used for the image restoration process is a two-dimensional filter, providing image restoration filters for respective image capturing states (image capturing conditions) such as focal lengths, object distances and aperture values and image heights enormously increases amount of data to be stored. However, Japanese Unexamined Patent Application Publication No. 2005-509333 does not mention reduction of the data amount.
Moreover, when performing the image restoration on the in-focus area of the captured image, since the in-focus area is changed in each image capturing, it is necessary to provide the image restoration filters for the entire image capturing area, which makes it difficult to reduce the data amount of the image restoration filters.